A transparent framework towards the context-sensitive recognition of conversational engagement

Modelling and recognising affective and mental user states is an urging topic in multiple research fields. This work suggests an approach towards adequate recognition of such states by combining state-of-the-art behaviour recognition classifiers in a transparent and explainable modelling framework that also allows to consider contextual aspects in the inference process. More precisely, in this paper we exemplify the idea of our framework with the recognition of conversational engagement in bi-directional conversations. We introduce a multi-modal annotation scheme for conversational engagement. We further introduce our hybrid approach that combines the accuracy of state-of-the art machine learning techniques, such as deep learning, with the capabilities of Bayesian Networks that are inherently interpretable and feature an important aspect that modern approaches are lacking - causal inference. In an evaluation on a large multi-modal corpus of bi-directional conversations, we show that this hybrid approach can even outperform state-of-the-art black-box approaches by considering context information and causal relations

I see what you did there: understanding when to trust a ML model with NOVA

In this demo paper we present NOVA, a machine learning and explanation interface that focuses on the automated analysis of social interactions. NOVA combines Cooperative Machine Learning (CML) and explainable AI (XAI) methods to reduce manual labelling efforts while simultaneously generating an intuitive understanding of the learning process of a classification system. Therefore, NOVA features a semi-automated labelling process in which users are provided with immediate visual feedback on the predictions, which gives insights into the strengths and weaknesses of the underlying classification system. Following an interactive and exploratory workflow, the performance of the model can be improved by manual revision of the predictions

This is not the Texture you are looking for! Introducing Novel Counterfactual Explanations for Non-Experts using Generative Adversarial Learning

With the ongoing rise of machine learning, the need for methods for explaining decisions made by artificial intelligence systems is becoming a more and more important topic. Especially for image classification tasks, many state-of-the-art tools to explain such classifiers rely on visual highlighting of important areas of the input data. Contrary, counterfactual explanation systems try to enable a counterfactual reasoning by modifying the input image in a way such that the classifier would have made a different prediction. By doing so, the users of counterfactual explanation systems are equipped with a completely different kind of explanatory information. However, methods for generating realistic counterfactual explanations for image classifiers are still rare. In this work, we present a novel approach to generate such counterfactual image explanations based on adversarial image-to-image translation techniques. Additionally, we conduct a user study to evaluate our approach in a use case which was inspired by a healthcare scenario. Our results show that our approach leads to significantly better results regarding mental models, explanation satisfaction, trust, emotions, and self-efficacy than two state-of-the art systems that work with saliency maps, namely LIME and LRP

ForDigitStress: presentation and evaluation of a new laboratory stressor using a digital job interview-scenario

IntroductionSince the COVID-19 pandemic, working environments and private lives have changed dramatically. Digital technologies and media have become more and more important and have found their way into nearly all private and work environments. Communication situations have been largely relocated to virtual spaces. One of these scenarios is digital job interviews. Job interviews are usually—also in the non-digital world—perceived as stressful and associated with biological stress responses. We here present and evaluate a newly developed laboratory stressor that is based on a digital job interview-scenario.MethodsN = 45 healthy people participated in the study (64.4% female; mean age: 23.2 ± 3.6 years; mean body mass index = 22.8 ± 4.0 kg/m2). Salivary alpha-amylase (sAA) and cortisol were assessed as measures for biological stress responses. Furthermore, perceived stress was rated at the time points of the saliva samplings. The job interviews lasted between 20 and 25 min. All materials, including instructions for the experimenter (i.e., the job interviewer) and the data set used for statistical analysis, as well as a multimodal data set, which includes further measures, are publicly available.ResultsTypical subjective and biological stress-response patterns were found, with peak sAA and perceived stress levels observed immediately after the job interviews and peak cortisol concentrations 5 min afterwards. Female participants experienced the scenario as more stressful than male participants. Cortisol peaks were higher for participants who experienced the situation as a threat in comparison to participants who experienced it as a challenge. Associations between the strength of the stress response with further person characteristics and psychological variables such as BMI, age, coping styles, and personality were not found.DiscussionOverall, our method is well-suited to induce biological and perceived stress, mostly independent of person characteristics and psychological variables. The setting is naturalistic and easily implementable in standardized laboratory settings

ForDigitStress: A multi-modal stress dataset employing a digital job interview scenario

We present a multi-modal stress dataset that uses digital job interviews to induce stress. The dataset provides multi-modal data of 40 participants including audio, video (motion capturing, facial recognition, eye tracking) as well as physiological information (photoplethysmography, electrodermal activity). In addition to that, the dataset contains time-continuous annotations for stress and occurred emotions (e.g. shame, anger, anxiety, surprise). In order to establish a baseline, five different machine learning classifiers (Support Vector Machine, K-Nearest Neighbors, Random Forest, Long-Short-Term Memory Network) have been trained and evaluated on the proposed dataset for a binary stress classification task. The best-performing classifier achieved an accuracy of 88.3% and an F1-score of 87.5%

MultiMediate'23: Engagement Estimation and Bodily Behaviour Recognition in Social Interactions

Automatic analysis of human behaviour is a fundamental prerequisite for the creation of machines that can effectively interact with- and support humans in social interactions. In MultiMediate'23, we address two key human social behaviour analysis tasks for the first time in a controlled challenge: engagement estimation and bodily behaviour recognition in social interactions. This paper describes the MultiMediate'23 challenge and presents novel sets of annotations for both tasks. For engagement estimation we collected novel annotations on the NOvice eXpert Interaction (NOXI) database. For bodily behaviour recognition, we annotated test recordings of the MPIIGroupInteraction corpus with the BBSI annotation scheme. In addition, we present baseline results for both challenge tasks.Comment: ACM MultiMedia'2

Drawn Stories, Moving Images. Comic Books and their Screen Adaptations

The comic transcends the merely entertaining, and fans of comics become engaged and invested in the field through a range of activities. Major cities host regular comic conventions, attracting hundreds of thousands of attendees each year, who search for special issues of their favourite comic-book series, meet artists, attend workshops and buy merchandise. Many fans do not stop at just attending conventions; they do so dressed as their favourite comic characters or wearing badges, buttons, T-shirts or sweaters with images of those characters on them. In other words: many fans do ot merely consume comic books; rather, they arrange a considerable part of their lives around them and in some cases even embody their heroes, that is, they copy their behaviour and their language. The comic universe, the comic books and the range of activities emerging out of them and around them become a meaningful universe for fans

Quantification of bulk lipid species in human platelets and their thrombin-induced release

Lipids play a central role in platelet physiology. Changes in the lipidome have already been described for basal and activated platelets. However, quantitative lipidomic data of platelet activation, including the released complex lipids, are unavailable. Here we describe an easy-to-use protocol based on flow-injection mass spectrometry for the quantitative analysis of bulk lipid species in basal and activated human platelets and their lipid release after thrombin activation. We provide lipid species concentrations of 12 healthy human donors, including cholesteryl ester (CE), ceramide (Cer), free cholesterol (FC), hexosylceramide (HexCer), lysophosphatidylcholine (LPC), lysophosphatidylethanolamine (LPE), phosphatidylcholine (PC), phosphatidylethanolamine (PE), phosphatidylinositol (PI), phosphatidylserine (PS), sphingomyelin (SM) and triglycerides (TG). The assay exhibited good technical repeatability (CVs < 5% for major lipid species in platelets). Except for CE and TG, the inter-donor variability of the majority of lipid species concentrations in platelets was < 30% CV. Balancing of concentrations revealed the generation of LPC and loss of TG. Changes in lipid species concentrations indicate phospholipase-mediated release of arachidonic acid mainly from PC, PI, and PE but not from PS. Thrombin induced lipid release was mainly composed of FC, PS, PC, LPC, CE, and TG. The similarity of the released lipidome with that of plasma implicates that lipid release may originate from the open-canalicular system (OCS). The repository of lipid species concentrations determined with this standardized platelet release assay contribute to elucidating the physiological role of platelet lipids and provide a basis for investigating the platelet lipidome in patients with hemorrhagic or thrombotic disorders